Greedy Structure Learning of Markov Random Fields

Acknowledgments I would like to thank my advisor, Pradeep Ravikumar, for inspiration, guidance, and encouragement on this work. In addition, I would like to thank Ali Jalali for his collaboration and work on the proof techniques and theoretical analysis used in this paper. Also, I would also like to thank Inderjit Dhillon and the students of his lab for motivation and many stimulating conversations like to thank my friends and family for their faith and encouragement in my many late nights of research and writing. I couldn't have finished this work without their support. Probabilistic graphical models are used in a variety of domains to capture and represent general dependencies in joint probability distributions. In this document we examine the problem of learning the structure of an undi-rected graphical model, also called a Markov Random Field (MRF), given a set of independent and identically distributed (i.i.d.) samples. Specifically, we introduce an adaptive forward-backward greedy algorithm for learning the structure of a discrete, pairwise MRF given a high dimensional set of i.i.d. samples. The algorithm works by greedily estimating the neighborhood of each node independently through a series of forward and backward steps. By imposing a restricted strong convexity condition on the structure of the learned graph we show that the structure can be fully learned with high probability given n = Ω(d log(p)) samples where d is the dimension of the graph and p is the number of nodes. This is a significant improvement over existing convex-optimization based algorithms that require a sample complexity of n = Ω(d 2 log(p)) and a stronger irrepresentability condition. We further v support these claims with an empirical comparison of the greedy algorithm to node-wise 1-regularized logistic regression as well as provide a real data analysis of the greedy algorithm using the Audioscrobbler music listener dataset. 2.1 Graphical representation of a Bayesian Network defined over a joint probability distribution. The " markov blanket " for the variable represented by the X consists of the shaded vertices.. 16 2.2 Graphical representation of a Factor Graph defined over a joint probability distribution. The circles represent random variables and the squares represent factors associated with the random 17 2.3 Graphical representation of a Markov Random Field (MRF) defined over a joint probability distritbution. The " markov blanket " or " neighborhood " of the variable represented by the X